Categorized data item reporting system and method

ABSTRACT

A system for managing resolvers of test answers to maintain consistent and efficient scoring of test answers. Collaborative scoring monitors and resolves discrepancies in two or more scores for the same test answers. Quality items monitor a test resolver&#39;s scoring performance against a known standard. Performance feedback data tracks a resolver&#39;s scoring rate to monitor efficiency and optionally recommend break times or provide the resolver with diversionary activities during breaks. Consistent scoring is further promoted by selectively matching resolvers with test answers based on classifications of the resolvers and persons who generated the test answers. An on-line scoring guide dynamically provides test resolvers with rules for scoring answers to particular test questions.

FIELD OF THE INVENTION

[0001] The present invention relates to a system for processing answersto test questions.

BACKGROUND OF THE INVENTION

[0002] The scoring of test answer sheets involves complex problems.These test answer sheets typically include a series of responsepositions such as, for example, “bubbles,” ovals, or rectangles. Aperson taking a test would, for example, darken in an appropriate ovalwith a pencil to answer a multiple choice question. These test answersheets may also include handwritten answers, such as essay or shortanswer questions. Systems for scanning and scoring the bubbles on suchanswer sheets are known in the art. Increased difficulties areencountered, however, when such answer sheets either include other typesof answers, such as handwritten answers, or cannot be machine graded.For example, if the student has failed to include his or her name on thetest answer sheet, the system may be unable to machine score the testanswer.

[0003] The goals in scoring test answers that cannot be machine scoredinclude efficiency and consistency. These test answer sheets aretypically scored by test resolvers either by manually scoring thephysical test answer sheet or scoring an electronic representation ofthe test answer sheet on a computer. Ideally, the scores provided by thevarious test resolvers for a particular test question should beconsistent, since the scores are used in comparing performance of thestudents against one another. In addition, a test resolver shouldideally work efficiently so as to maintain consistently high scoringrates. The test resolver should not have such a high scoring rate thatthe consistency or quality of scoring significantly declines; likewise,the test resolver should not have such a low scoring rate that the toofew answer sheets are being scored. This manual scoring of test answersheets, however, makes it difficult to monitor the consistency ofscoring among the various test resolvers.

[0004] In many situations, test resolvers actually travel to aparticular location so that all test resolvers may simultaneously scoretest answer sheets. Requiring the test resolvers to travel to a givenlocation is inconvenient for the resolvers and expensive for those whoadminister the tests. Furthermore, tracking the performance of testresolvers against both their own performance and the performance ofother resolvers can be very difficult with a manual scoring environment.

[0005] The process of resolving test questions is currently donemanually, and this presents problems. A resolver is manually presentedwith the actual test answer sheets for scoring. This process isrelatively inefficient, since the resolvers must score the answer sheetsone at a time and in the order in which they are presented. Also, manualscoring systems do not have the capability to efficiently gather andcategorize the test answers for subsequent analysis. Therefore, with amanual system it is very difficult to determine how teaching methodsshould be changed to decrease, for example, the number of incorrectanswers.

[0006] A need thus exists for a system that promotes and achievesconsistency and efficiency in scoring or resolving of tests.

SUMMARY OF THE INVENTION

[0007] The present test resolver management system facilitatesconsistent, accurate, and high quality scoring of test answers. Acollaborative scoring sub-system allows two test resolvers to score thesame test answer and resolves potential discrepancies in the scoresprovided by the resolvers.

[0008] The system also has the capability to use quality items tomonitor a resolver's scoring performance. The system transmits aquality-control test item, having a known score, to a test resolver.After receiving a score from the resolver, the system compares the knownscore with the resolver's score and can generate a quality profile forthe resolver.

[0009] The system also provides performance feedback to test resolversby monitoring the scoring rates of the resolvers. If a resolver'sscoring rate, in comparison to another scoring rate, contains adiscrepancy, the system can report the discrepancy. The system can alsorecommend breaks to a resolver, or provide a resolver with diversionaryactivities, in response to a discrepancy in scoring rates. The systemalso has the capability to route test answers to resolvers based onclassifications of resolvers and persons who generated the test answers.Monitoring and selecting resolvers based on classifications promotesfair and consistent scoring of test answers.

[0010] Finally, the system provides for an on-line scoring guide todynamically provide a test resolver with scoring rules for a test answerdisplayed to the resolver.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011]FIG. 1 is a diagram of a network that incorporates the presentinvention.

[0012]FIG. 2 is a block diagram of a portion of the network shown inFIG. 1.

[0013]FIG. 3 is a block diagram of the scanning configuration in thenetwork of FIG. 1.

[0014]FIG. 4 is a block diagram of the server in the network of FIG. 1.

[0015]FIG. 5 is a flow chart of receiving and processing of test items.

[0016]FIG. 6 is a flow chart of multiple item scoring.

[0017]FIG. 7 is a flow chart of categorized (special) item reporting.

[0018] FIGS. 8-10 are a flow chart of collaborative scoring.

[0019]FIG. 11 is a flow chart of quality item processing.

[0020]FIG. 12 is a flow chart of resolver monitoring and feedback.

[0021]FIG. 13 is a flow chart of an on-line scoring guide system.

[0022]FIG. 14 is an example of a user interface for use with multipleitem scoring.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0023] In the following detailed description of the preferredembodiment, reference is made to the accompanying drawings which form apart hereof and in which is shown by way of illustration a specificembodiment in which the invention may be practiced. This embodiment isdescribed in sufficient detail to enable those skilled in the art topractice the invention, and it is to be understood that otherembodiments may be utilized and that structural or logical changes maybe made without departing from the scope of the present invention. Thefollowing detailed description is, therefore, not to be taken in alimiting sense, and the scope of the present invention is defined by theappended claims.

Hardware Configuration

[0024]FIG. 1 illustrates an example of a hardware configuration for anetwork that incorporates the present invention. This configuration isshown as an example only; many different hardware configurations areavailable, as recognized by one skilled in the art, for implementing thesoftware processing functions described below. The network showncomprises a mainframe computer 20 interfaced through a backbone tokenring to a plurality of RISC servers 11, 12 and 13. Each RISC server isinterfaced to a token ring that contains work stations and scanners. TheRISC server 11 is connected in token ring 1 to scanners 14 and workstations 19. The RISC server 12 is connected in token ring 2 to scanner15 and work stations 18. The RISC server 13 is connected in token ring 3to scanners 16 and work stations 17. The mainframe computer 20 is alsoconnected to a high capacity printer 22 and a low capacity printer 21for printing reports of stored data within the system.

[0025] The system uses the scanners for reading in test answer sheets.These test answer sheets may comprise, for example, test forms with“bubbles” or ovals representing possible answers, handwritten essays, orother various types of written or printed information. After receivingthe scanned test data, the system within the RISC servers can processthose scanned test answer sheets to generate test items of interest fromthe answer sheets. A test item is, therefore, an electronicrepresentation of at least a portion of a test answer sheet. The systemmay distribute these test items to the work stations for on-linescoring. A test scorer at a work station can then score the test itemand enter a test score. The system receives the test scores via thenetwork and the RISC servers and distributes the scores to anappropriate computer for subsequent printing and reporting; theappropriate computer may include, for example, the mainframe computer 20or a server. The system may also transmit the test scores to, forexample, a disk or telephone line.

[0026]FIG. 2 is a more detailed block diagram of a portion of thenetwork shown in FIG. 1. As shown in FIG. 2, the scanning units shown inFIG. 1 typically comprise a scanner 25 interfaced to a computer 24 andpersonal computer (PC) 26. FIG. 3 shows a more detailed block diagram ofa scanning unit. The scanner 25 contains a camera 31 for opticallyreading in a test answer sheet, and further contains optical markrecognition (OMR) logic 32 for processing the scanned data received fromcamera 31. The PC 26, preferably implemented with a high performance486-level PC, contains a frame buffer 23 for receiving the scanned imagedata from the scanner 25.

[0027] The computer 24, preferably implemented with an HP 1000, isinterfaced to the scanner 25 and PC 26 for controlling the operation ofthe scanning unit. The computer 24 is optional; the system mayalternatively be configured such that all of the functionality of thecomputer 24 is within the PC 26. The computer 24 controls the scannervia the OMR logic 32 and thus controls when image data is scanned in andsubsequently transferred to the PC 26. The PC 26 essentially acts as abuffer for holding the image data. The computer 24 further controls whenthe PC 26 will interrogate the image data for transmission to a server27 for subsequent processing and scoring. The PC 26 can alsoelectronically remove or “clip” an area of interest from the image data,which represents at least a portion of the scanned test answer sheets.

[0028] Examples of two systems for storing and extracting informationfrom scanned images of test answer sheets are shown in U.S. Pat. Nos.5,134,669 and 5,103,490, both of which are assigned to National ComputerSystems, Inc. and are incorporated herein by reference as if fully setforth.

[0029] The server 27 receives the image data, which includes test items,and provides for processing and control of the image data. This portion,which may be a test item, is then distributed to the work stations 28,29 and 30 for subsequent scoring. A test resolver (scorer) at the workstation typically receives the test item, performs the scoring, andtransmits the score to the receiving computer.

[0030]FIG. 4 is a block diagram of the hardware and software functionsin a server in the network of FIG. 1. A scan control module 31interfaces with the scanner PC 26 and receives the image data. The imagedata is stored in a raw item database 36. The central applicationrepository (CAR) 33 typically stores document definitions and handlingcriteria. The document process queue 37 functions as a buffer into amain processing module 45 in server 27.

[0031] The main processing module 45 controls the processing of testitems. It controls the transmission of test items to the work stationsfor scoring and the transmission of scores to the mainframe computer 20.The main processing module 45 also monitors the performance of the testresolvers to maintain consistent and efficient resolving of test items,as is explained below.

[0032] The main processing module 45 typically contains the followingbasic functions, which are controlled by system management module 32. Awork flow module 38 receives image data from the database 36 andcontrols the flow of data into an edit process module 39. The editprocess module 39 may perform machine scoring of the test items. Forthose test items which cannot be machine scored, or possibly for othertest items, the system transmits such test items to the job buildfunction 40. The job build function 40 determines what type ofsubsequent scoring is required for the test item and, for example, whichwork station will receive the test item. A job send module 41 receivesthe test item and transmits it to a router 42, which in turn transmitsthe test item to a send/receive communication module 43. Edit workmodule 34 and edit server module 35 control the flow of test items intoand out of server 27. Incoming data, such as test answers from the workstation, are transmitted through modules 34 and 35 to a job receivemodule 44. The job receive module transmits the data to the edit processmodule 39 for subsequent storage within the database 36.

Softwear Processing

[0033]FIG. 5 is a flow chart of typical scanning and processing of testand answer sheets. The document processing system receives the testanswer sheets, or other documents, at step 50 and performs initialclerical preparation of the documents (step 51) for scanning at step 52.The system at step 52 scans the documents for OMR and other image data.The system may then process the OMR bubbles at step 53 and store thedata in the work-in-process storage (WIP) at step 54. The system at step56 can “clip” areas of interest from the scanned image. The step of“clipping” involves electronically removing, typically in software, aportion of the test item or scanned image. These “clipped” areas maycomprise any portion of a test answer sheet; for example, a handwrittenessay or selected response positions. The system may also receive imagedata directly from foreign sources, magnetic or electronic, and storethe data in raw item database 36. Subsequent operations on the data arethe same regardless as to the source of the data. After “clipping” areasof interest from the image, the system stores the test items at step 57in the work-in-process storage 55.

[0034] The system waits at step 55 until it determines that a test itemis ready to be resolved or scored. If multiple resolution items arepresent within the image data, as determined at step 59, then the systemsends the test item to multiple item processing at step 63. Otherwise,the system performs other resolution processes on the data at step 60and stores the result in work-in-process storage 55 at step 61. Otherresolution processes may include, for example, machine scoring, raw keyentry, and analytic resolving.

[0035] Analytic resolving or scoring may include, for example, mapcomparisons such as bit-mapped comparisons between two test items. Themap comparisons allow a test resolver to compare, for example, theanswers of a respondent over time to track the respondent's progress.For example, the analytic scoring may involve comparing two hand-drawncircles by the respondent to determine if the respondent's accuracy indrawing circles has improved over time. Analytic scoring may alsoinclude, for example, circling or electronically indicating misspelledwords and punctuation errors in an answer such as an essay.

Multiple Item Scoring

[0036]FIG. 6 is a flow chart of typical multiple item processing. Thesystem at step 64 typically first fetches a multiple item image from thework-in-process storage. The image is stored in a multiple item displaymemory 65 and a multiple item display storage 69 for subsequent displayto a resolver. The system continues to receive multiple items untileither the item display is full, as determined at step 66, or no moremultiple items are present as determined at step 68. As long as thedisplay is not full and additional multiple items are present, thesystem preferably scans the work-in-process storage at step 67 foradditional items. When the multiple item display is full or no moremultiple items are present, the system sends the compiled multiple itemsto a resolver at step 70 and displays the multiple test items on theresolver display 71.

[0037] The system typically transmits test items to a particularresolver based upon the resolver's resolution expertise. For example, acertain resolver may be assigned to score all of the test items relatingto science questions. Resolution expertise may also comprise, forexample, math, english, history, geography, foreign languages, or othersubjects.

[0038] An example of an interface on the resolver display is shown inFIG. 14. The interface typically comprises a plurality of cells 74, witheach cell containing one test item to be resolved. After displaying themultiple items in the cells of the resolver display, the system allowsthe resolver at step 72 to score the multiple items. A test resolverwould typically indicate the score of the answers by using a “mouse,”light pen, touch screen, voice input, or some other type of cursorcontrol or input device.

[0039] In the example shown in FIG. 14, the correct answer is “four” andthe incorrect answers are indicated by the shading. Alternatively, aresolver could indicate the correct answers. The advantage of themultiple item system arises from the simultaneous display of test itemsin the cells 74, which allows a test resolver to quickly score many testitems and thus achieve a faster response time in comparison to thedisplay and scoring of only a single test item at a time. Even thesimultaneous display of two items increases response time. As the matrixof cells increases, the simultaneous display of test items achieves asignificant increase in response time and resolver attention and focus.

[0040] After scoring or resolving, the system receives the results atstep 73 for subsequent storage in work-in-process storage 55. A testresolver typically transmits the results of resolving all displayed testitems in the cells as a single unit for batch processing.

Categorized Item Reporting

[0041]FIG. 7 is a typical flow chart of categorized (special) itemreporting. Categorized item reporting allows the system to both groupanswers according to predefined categories and monitor processes used bythe students or test-takers in arriving at a given answer. Thecategories in which test answers may be grouped include, for example,incorrect answers and correct answers within a curriculum unit within aninstructional grouping and requested time frames; for example, all ofthe incorrect math answers in a particular instructor's class during theprevious school year. Other groupings are possible depending upon theneeds of the test resolvers and instructors who teach the material towhich the test relates.

[0042] In addition, the system may merge an image of a test item withthe corresponding score. In order to facilitate teaching of material towhich the test relates, the system typically merges a test itemrepresenting an incorrect answer with the corresponding score. Byreporting the actual test item, an instructor may gain insight into athought process used by the student in arriving at the incorrect answer.Therefore, by having some knowledge of why a student answered a testquestion incorrectly, an instructor can take measures to change ormodify teaching strategies to correct the situation.

[0043] The categorized item reporting normally comprises the followingfunctions. The system at step 75 scans the work-in-process storage foritems that are ready to be reported. If test items are ready forreporting, as determined at step 76, the system processes the data atstep 77 for generating an appropriate report of the data. At step 78,the system scans the central application. repository for definitions ofcategorized (special) items. As special items are available forreporting, as determined at step 79, the system retrieves the specialitems at step 80 and can merge it at step 81 with other reportinformation, such as the corresponding test items, as explained above.The system then distributes a report at step 82, which can be a printedreport.

Collaborative Scoring

[0044] FIGS. 8-11 are a flow chart of a typical collaborative scoringsystem. The collaborative scoring system provides for functions toachieve fairness and objectivity in resolving of test items. Thecollaborative scoring, for example, allows two resolvers to score thesame item and, if the answers are not within a certain predefined range,provides for subsequent processing to resolve the discrepancy.

[0045] The system at steps 83 and 84 determines if items are availablefor scoring. At step 85, the system receives collaborative scoringrequirements from the database and determines at step 86 ifcollaborative scoring is required. Examples of collaborative scoringrequirements are illustrated below. If collaborative scoring has beenspecified, the system retrieves the item to be scored from thework-in-process database at step 87 and sends the item to resolvers 1and 2 at steps 88 and 91.

[0046] The system is further able to choose resolvers according toselection criteria at steps 89 and 90. The selection criteria of theresolvers for scoring answers may include, for example, race, gender, orgeographic location. The ability of the system to assign test resolversto score particular test items provides the basis for increased fairnessand consistency in the scoring of tests. For example, the system mayassign test resolvers to test items based on the same racialclassification, meaning that the test resolver has the same racialclassification as the student or respondent whose test the resolver isscoring. The system may also assign test resolvers to test items basedon a different, forced different, or preferred blend of classifications.The system monitors consistency in scoring based on the selectioncriteria and, more importantly, can change the selection criteria toensure consistent and fair scoring of test items.

[0047]FIG. 9 is a flow chart showing additional typical functions of thecollaborative scoring. At steps 92 and 93, the system displays the itemsto resolvers 1 and 2 for scoring. The system may further track theaverage scores of resolvers and not send the same test item to tworesolvers who have provided average scores within a predefined range.This also helps to achieve consistency in scoring. For example, if twoscorers each have provided high average scores in the past, asdetermined by the system, these two scorers should preferably not becollaboratively scoring the same test items, since it could result in“inflated” scores for particular test items.

[0048] The system records the scores from resolvers 1 and 2 at steps 94and 95, respectively, and stores such scores in a temporary storage 96.At step 97, the system compares the scores according to criteriaspecified in the central application repository. Such criteria mayinclude, for example, requiring that the scores be within a predefinedpercentage of each other. If the scores meet the criteria as determinedat step 98, the system records the score in the work-in-process databaseat step 46. Otherwise, if the scores do not meet the criteria, thesystem determines at step 99 if the scores of the resolvers must agree.If the first two resolvers scores do not need to agree, then the systempreferably transmits the test item to a third resolver to “cure” thediscrepancy in the first two scores. At step 100, the system determinesif the third resolver should see the first scores.

[0049]FIG. 10 shows additional typical processing of the collaborativescoring. If the original resolvers 1 and 2 must agree on a score, thenthe system executes steps 101-105. The system then typically firstdisplays to each resolver the other resolver's score at steps 101 and102 so that each resolver can see the score determined by the otherresolver. At step 103, the system establishes a communication betweenthe two resolvers. Such a communication link may be, for example, anelectronic mail link so that the scorers can exchange informationregarding the reasoning behind the score provided. At step 104, theresolvers work together to determine a single agreed-upon score for thetest item. The system may prevent the resolvers 1 and 2 from receivinganother test item until they have entered an agreed-upon score for theprevious test item. Finally, at step 105, the system stores theagreed-upon score in the work-in-process database.

[0050] Instead of allowing the resolvers to work together to record anagreed-upon score, the system may optionally record either a greatervalue of the first and second test scores, a lower value of the firstand second test scores, or an average value of the first and second testscores.

[0051] If the collaborative scoring criteria specifies that the thirdresolver should arbitrate the discrepancy and determine a score, thenthe system displays scores from the resolvers 1 and 2 at step 106 forresolver 3. The third resolver (resolver 3) then typically enters ascore for the test item at step 107, and the system records the score inthe work-in-process database at step 108.

[0052] If the collaborative scoring requirement specifies that the thirdresolver should not see the first two scores, then the system executessteps 109-111. At step 109, the system displays the test item for thethird resolver. The third resolver then typically enters a score at step110, and the system records the score in the work-in-process database atstep 111.

Quality Items

[0053]FIG. 11 is a typical flow chart of the use of quality items in thescoring process. The system can use quality items to check and monitorthe accuracy of the scoring for selected test resolvers in order tomaintain consistent and high quality scoring of test items. At step 112,the system determines or receives the quality criteria. The qualitycriteria may be, for example, a predetermined test item with a known“correct” score.

[0054] The system then waits for a scheduled quality check at step 113.At the quality check, the system, at step 114, sends the known qualityitem to the scheduled resolver. At step 116, the system updates theresolver's quality profile based on the evaluation at step 115. If theresolver should receive a quality result, as determined at step 117, thesystem displays the quality profile to the resolver at step 118. At step119, the system sends the quality profile to a manager for subsequentreview. At step 120, the system takes action required to assure scoringaccuracy.

Resolver Monitoring and Feedback

[0055]FIG. 12 is a flow chart of typical resolver monitoring andfeedback. The primary factors in monitoring performance typicallyinclude: (1) validity; (2) reliability; and (3) speed. In monitoringthese factors, the system promotes repeatability of scoring. Thesefactors may be monitored by tracking a resolver's performance againstpast performance of the resolver or against some known goal.

[0056] Validity is typically measured by determining if a particularresolver is applying the scoring key correctly to test items or, inother words, scoring test items as an expert would score the same items.Reliability is typically measured by determining if a particular willresolve the same test item the same way over time (providing consistentscoring). Speed is typically measured by comparing a resolver's scoringrate with past scoring rates of the resolver or other scoring rates,such as average scoring rates or benchmark scoring rates.

[0057] At step 121, the system typically continually monitors theresolver's performance and updates the performance. Monitoring theresolver's performance may include, as explained above, monitoring theresolver's validity, reliability, and speed in resolving test items. Thesystem periodically, according to predefined criteria, performsperformance checks of the test resolvers. Predefined criteria mayinclude, for example: a time period; recalls (how often a resolverevaluates his or her own work); requesting help; the number ofagreements among multiple resolvers; the amount of deviation between theresolver's score and a known score, which may be determined usingquality items; the frequency of these deviations; the speed at which aresolver enters a response during resolving of test items; the length oftime between scores entered by a test resolver; a test resolver'sprevious scoring rate, an average scoring rate of a test resolver;average scoring rates of other test resolvers; or some predeterminedbenchmark scoring rate.

[0058] At step 122, the system determines whether it is time for ascheduled performance check according to the predetermined criteria. Ifit is time for a performance check, the system at step 123 compares theresolvers' current performance, as determined at step 121, with thestored performance criteria. At step 124, the system determines if thereis a discrepancy in the resolver's performance according to thepredetermined criteria. For example, the system may determine if theresolver's current scoring rate is within a predefined percentage of theaverage scoring rate in order to ensure efficient scoring by the testresolver. If there is no discrepancy, the system returns to monitoringthe resolver's performance. In addition, the system may store theresolver's current performance values for later processing. Otherwise,the system reports the discrepancy at step 125.

[0059] At step 126, the system determines if it should recommend a breakin scoring to the resolver. If according to predetermined performancecriteria, the system should recommend a break in scoring, then thesystem signals the resolver at step 128 to halt scoring. Predefinedperformance criteria may include, for example, deviations in theresolver's validity, reliability, or speed of resolving test items.Examples of predefined performance criteria are provided above withrespect to the monitoring of resolvers' performance.

[0060] When the resolver stops scoring, the system may provide theresolver with the option of requesting diversionary activities.Diversionary activities are designed to provide the test resolver with arest period and “break” from scoring to increase efficiency. Examples ofdiversionary activities include computer games and cross word puzzles.If the resolver has requested such diversionary activities, asdetermined at step 129, then the system transmits a diversionaryactivity to the resolver at step 130. Otherwise, the system returns tomonitoring the resolver's scoring rate when the resolver resumes thescoring.

[0061] If the system at step 126 does not recommend a break in scoringbased on the discrepancy, then the system may optionally provide theresolver with diversionary activities as determined at step 127. If theresolver should receive the diversionary activities, then the systemsends such activities to the resolver at step 130. Otherwise the systemreturns to monitoring the resolver's scoring rate.

On-Line Scoring Guide

[0062]FIG. 13 is a flow chart of a typical on-line scoring guide system.The on-line scoring guide increases scoring efficiency by allowing theresolver to request scoring rules in order to assist in scoring aparticular test item. In response to the request, the system displaysscoring rules corresponding to a test question for the test itemcurrently displayed to the resolver. A resolver may thus quickly havespecific scoring rules available on-line while scoring test items. Thispromotes scoring efficiency and reduces unnecessary break timesresulting from determining how to score a particular test item.

[0063] At step 131, the system sends a test item to a resolver forscoring and displays the test item at step 132. If the resolver hasrequested scoring rules, as determined at step 133, then the systeminterrogates a stored scoring guide to locate scoring rules thatcorrespond to a test question for the test item currently displayed tothe resolver. The system retrieves those particular scoring rules atstep 135 and displays them to the resolver at step 136. The systempreferably uses a multi-tasking environment in order to simultaneouslydisplay the scoring rules and the test item. At step 134, the systemwaits for the resolver to score the test item. At step 137, the systemstores the test score entered by the resolver into the work-in-processstorage.

[0064] As described above, the present invention is a system thatprocesses test items. The various functions used in processing the testitems promote efficient, high quality, and consistent scoring of testitems.

[0065] While the present invention has been described in connection withthe preferred embodiment thereof, it will be understood that manymodifications will be readily apparent to those skilled in the art, andthis application is intended to cover any adaptations or variationsthereof. For example, a different hardware configuration may be usedwithout departing from the scope of the invention and many variations ofthe processes described may be used. It is manifestly intended that thisinvention be limited only by the claims and equivalents thereof.

What is claimed is:
 1. A system for verifying test answers and providingcollaborative scoring of the test answers between two or more testresolvers if the test answers contain a discrepancy, the systemcomprising: a) first receive means for electronically receiving ananswer to a test question, the answer comprising an electronicrepresentation of at least a portion of a test answer sheet; b) transmitmeans for electronically and independently transmitting the answer tofirst and second test resolvers; c) second receive means forelectronically receiving first and second test scores from the first andsecond test scorers; and d) verification means for electronicallycomparing the first and second test scores and for selectively recordinga value related to the first and second test scores if the first andsecond test scores did not contain a discrepancy based on predefinedcriteria.
 2. The system of claim 1 further comprising display means forelectronically displaying the test question.
 3. The system of claim 1wherein the verification means further comprises collaboration means forproviding an electronic communication link between the first and secondtest resolvers and, if the first and second scores contain thediscrepancy based on the predefined criteria, for signalling the firstand second test resolvers to interactively determine and enter a jointscore, the verification means further comprising recording means forrecording the joint score.
 4. The system of claim 3 wherein thecollaboration means further comprises means for tracking average scoresentered by the first and second resolvers and for preventing the firstand second resolvers from scoring the same test item if the first andsecond resolvers each have average scores within predefined scoringcriteria.
 5. The system of claim 3 wherein the verification meansfurther comprises means for preventing the first and second testresolvers from receiving another answer to score until the first andsecond test resolvers have entered the joint score.
 6. The system ofclaim 3 wherein the verification means comprises third resolver meansfor transmitting the test item to a third test resolver if the first andsecond test scorers do not enter the joint score within a predeterminedtime period.
 7. The system of claim 1 wherein the verification meansfurther comprises third resolver means for transmitting the answer to athird test resolver, if the first and second scores contain thediscrepancy based on the predefined criteria, and for recording a thirdtest score received from the third test resolver.
 8. The system of claim7 wherein the third resolver means further comprises means for providingthe third test resolver with the first and second scores.
 9. The systemof claim 7 wherein the third resolver means further comprises means forpreventing the third test resolver from viewing the first and secondscores before the third test resolver records the third test score. 10.The system of claim 1, wherein the verification means further comprisesmeans for electronically linking the first and second test resolverswhen the first and second test resolvers are located in differentgeographical locations.
 11. The system of claim 1 wherein theverification means comprises means for recording at least one of agreater value of the first and second test scores, a lower value of thefirst and second test scores, and an average value of the first andsecond test scores.
 12. A system for evaluating the performance of testresolvers by routing quality-control questions for scoring to testresolvers, the quality-control questions comprising a test answer with ascore known to quality-control personnel, the system comprising: a)transmit means for electronically transmitting to a test resolver anunidentified quality-control question comprising a test answer with ascore known to quality-control personnel; b) receive means forelectronically receiving from the test resolver a score for thequality-control question; and c) processing means for electronicallycomparing the score known to the quality-control personnel and theresolver score and for reporting a discrepancy between the known scoreand the resolver score.
 13. A system for electronically providingperformance feedback based upon a comparison of a test resolver'sperformance to predetermined performance criteria, the systemcomprising: a) receive means for electronically receiving informationrelated to a first performance level from a first resolver and apredetermined performance level; b) comparison means for electronicallycomparing the first performance level and the predetermined performancelevel; and c) feedback means for electronically reporting a discrepancybased on predefined criteria between the first performance level and thepredetermined performance level.
 14. The system of claim 13 wherein thefeedback means comprises means for electronically reporting thediscrepancy to the first test resolver.
 15. The system of claim 13wherein the feedback means comprises means for electronically reportingthe discrepancy to a system manager.
 16. A system for electronicallyproviding performance feedback based upon a comparison of a testresolver's performance to predetermined performance criteria, the systemcomprising: a) receive means for electronically receiving informationrelated to a first test-score validity level from a first resolver and apredetermined validity level; b) comparison means for electronicallycomparing the first validity level and the predetermined validity level;and c) feedback means for electronically reporting a discrepancy basedon predefined criteria between the first validity level and thepredetermined validity level.
 17. The system of claim 16 wherein thereceive means comprises means for receiving a test score for a qualityitem from the first resolver and a known score for the quality item. 18.The system of claim 16 wherein the feedback means comprises means forelectronically reporting the discrepancy to the first test resolver. 19.The system of claim 16 wherein the feedback means comprises means forelectronically reporting the discrepancy to a system manager.
 20. Asystem for electronically providing performance feedback based upon acomparison of a test resolver's performance to predetermined performancecriteria, the system comprising: a) receive means for electronicallyreceiving information related to a first reliability level from a firstresolver and a predetermined reliability level; b) comparison means forelectronically comparing the first reliability level and thepredetermined reliability level; and c) feedback means forelectronically reporting a discrepancy based on predefined criteriabetween the first reliability level and the predetermined reliabilitylevel.
 21. The system of claim 20 wherein the receive means comprisesmeans for receiving during a first time period a first score from thefirst resolver for a test item and for receiving during a second timeperiod a second score from the first resolver for the test item.
 22. Thesystem of claim 20 wherein the feedback means comprises means forelectronically reporting the discrepancy to the first test resolver. 23.The system of claim 20 wherein the feedback means comprises means forelectronically reporting the discrepancy to a system manager.
 24. Asystem for electronically providing performance feedback based upon acomparison of a test resolver's performance to predetermined performancecriteria, the system comprising: a) receive means for electronicallyreceiving information related to a first speed level from a firstresolver and a predetermined speed level; b) comparison means forelectronically comparing the first speed level and the predeterminedspeed level; and c) feedback means for electronically reporting adiscrepancy based on predefined criteria between the first speed leveland the predetermined speed level.
 25. The system of claim 24 whereinthe receive means comprises means for receiving at least one of thefollowing: a first scoring rate for the first resolver, an averagescoring rate from a selected group of resolvers, and a second scoringrate from a second resolver.
 26. The system of claim 24 wherein thefeedback means comprises means for electronically reporting thediscrepancy to the first test resolver.
 27. The system of claim 24wherein the feedback means comprises means for electronically reportingthe discrepancy to a system manager.
 28. A system for increasing testresolver efficiency by monitoring test resolver performance andrecommending break times to the test resolvers, the system comprising:a) receive means for electronically receiving at predetermined pastintervals in time first performance levels from a test resolver and forelectronically receiving a second predetermined performance level; b)comparison means for electronically comparing the second predeterminedperformance level and the first performance level; and c) recommendationmeans for electronically detecting a discrepancy based on predefinedcriteria between the second predetermined performance level and thefirst performance level as indicated by the comparison means and forsignalling the test resolver to halt test scoring for a break time. 29.The system of claim 28 wherein the receive means comprises means forelectronically receiving first validity levels from the test resolverand for electronically receiving a second predetermined validity level.30. The system of claim 28 wherein the receive means comprises means forelectronically receiving first reliability levels from the test resolverand for electronically receiving a second predetermined reliabilitylevel.
 31. The system of claim 28 wherein the receive means comprisesmeans for electronically receiving first speed levels from the testresolver and for electronically receiving a second predetermined speedlevel.
 32. The system of claim 28 wherein the recommendation meansfurther comprises diversion means for electronically detecting adiscrepancy based on predefined criteria between the second scoring rateand the first scoring rates as indicated by the comparison means and forproviding the test resolver with diversionary activities.
 33. The systemof claim 28 wherein the recommendation means further comprisesresolver-selected diversion means for enabling the test resolver toselect diversionary activities when the recommendation means hassignalled the test resolver for the break.
 34. A system for increasingtest resolver efficiency by monitoring test resolver performance andrecommending break times to the test resolvers, the system comprising:a) receive means for electronically receiving at predetermined pastintervals in time first test scoring rates from a test resolver and forelectronically receiving a second present scoring rate from the testresolver; b) comparison means for electronically comparing the secondscoring rate and the first scoring rates; and c) recommendation meansfor electronically detecting a discrepancy based on predefined criteriabetween the second scoring rate and the first scoring rates as indicatedby the comparison means and for signalling the test resolver to halttest scoring for a break time.
 35. The system of claim 34 wherein therecommendation means further comprises diversion means forelectronically detecting a discrepancy based on predefined criteriabetween the second scoring rate and the first scoring rates as indicatedby the comparison means and for providing the test resolver withdiversionary activities.
 36. The system of claim 34 wherein therecommendation means further comprises resolver-selected diversion meansfor enabling the test resolver to select diversionary activities whenthe recommendation means has signalled the test resolver for the break.37. A system for preventing bias in the scoring of answers to teststaken by persons in different particular classifications, the systemcomprising: a) receive means for electronically receiving a plurality ofanswers to test questions, the test answers each corresponding to a testtaken by a person from a particular definable classification andcomprising an electronic representation of at least a portion of one ormore test answer sheets; b) classification means for receiving aparticular classification of each of the respondents or a group of therespondents and for electronically associating each of the particularclassifications with the test answers for the corresponding respondentor group; and c) routing means for electronically selecting a testresolver for each of the answers based upon the particularclassification of the corresponding respondent or group and fortransmitting the answers to the selected test resolver.
 38. The systemof claim 37 wherein: a) the classification means comprises means forreceiving a racial classification identifying a race of each of the testresolvers; and b) the routing means comprises means for electronicallytransmitting each of the test answers to a test resolver having a same,different, or preferred blend racial classification as the racialclassification of the corresponding respondent or group.
 39. The systemof claim 37 wherein: a) the classification means comprises means forreceiving a geographical classification identifying a location of eachof the test resolvers; and b) the routing means comprises means forelectronically transmitting each of the answers to a test resolverhaving a same, different, or preferred blend geographical classificationas the geographical classification of the corresponding respondent orgroup.
 40. A system for electronically and dynamically providing scoringrules to test resolvers to assist the test resolvers in scoring testitems to which the scoring rules apply, the system comprising: a)receive means for electronically receiving a plurality of answers fortest questions, the answers each comprising an electronic representationof at least a portion of a test answer sheet; b) storage means forelectronically storing a plurality of scoring rules, each of the scoringrules relating to a procedure for scoring at least a particular one ofthe test questions; c) display means for selectively presenting the testanswers to a test resolver; and d) rules means for selectivelypresenting to the test resolver the scoring rules related to the testquestion corresponding to the answer presented to the test resolver.